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Abstract. Let G be a simple real Lie group with maximal parabolic subgroup 
P whose nilradical is abelian. Then X = G/P is called a symmetric /?-space. 
We study the degenerate principal series representations of G on C°° (X) in the 
case where P is not conjugate to its opposite parabolic. We find the points of 
reducibility, the composition series and all unitarizable constituents. Among 
the unitarizable constituents we identify some small representations having 
as associated variety the minimal nilpotent /fc-orbit in pj, where Kc is the 
complexification of a maximal compact subgroup K G G and g = t + p the 
corresponding Cartan decomposition. 



Introduction 

Let X ~ G/P be an irreducible symmetric i?-space, i.e. G is a simple real non- 
compact Lie group and P a maximal parabolic subgroup with abelian nilradical. 
Then X is at the same time a compact symmetric space X = K/M where K C G 
is a maximal compact subgroup of G and M — K D P. Write P = MpApN 
for the Langlands decomposition of P and let ap be the Lie algebra of Ap. On 
X we consider the degenerate principal series representations (normalized smooth 
parabolic induction) 

I{i^) Ind^(l ® e"^ ® 1) C G°°{X) 

of parameter i> G (cip)J. In this paper we are concerned with the structure of these 
representations. Natural questions in this framework are: 

(1) For which i/ G (ctp)c is irreducible? 

(2) What is the composition series of a reducible /(z^)? 

(3) Which irreducible constituents are unitary? 

A key role in the study of these questions is played by intertwining operators. In 
particular, a G-invariant subspace W C /(i/) is unitary if and only if there exists 
an intertwining operator T : W with strictly positive eigenvalues. In this 

case the inner product on W given by 



W xW ^<C,{f,g)^ I f{x)Tg{x)dx 
Jx 



is G-invariant. 
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Under the assumption that P is conjugate to its opposite parabohc subgroup 
P the questions (1), (2) and (3) have been completely answered by Johnson |20] . 
0rsted-Zhang ^5] j Sahi and Zhang [IS] . This case occurs exactly when 

there exists a Weyl group element for the restricted root system A of g (with respect 
to a maximally non-compact Cartan) which acts on ap by —1. Hence the classical 
Knapp-Stein theory provides standard intertwining operators A{v) : J(i/) — > li—v) 
with meromorphic dependence in the parameter v ^ (ap)^. In the above mentioned 
papers these Knapp-Stein intertwiners and their residues are used to treat the above 
questions. 

In this paper we completely answer the above questions for the case where P 
is not conjugate to P (see Tables [T] and [5] for a classification). Up to covering the 
possible groups G, sorted by the restricted root system A of g with respect to a 
maximally non-compact Cartan, and the corresponding Lie algebras I of the Levi 
part L = MpAp of P ^ LN are 

• Type A 

- G = SL{r + s,K) with [ = s(fll(r, IK) ® fll(s, K)) and IK e {R,C,IHI}, 

- G = SL(3, 0) = Eqi^_26) with I = so(l, 9) ® R, 

• Type D 

- G:=SOo(2r+l,2r-hl) with [ = 0[(2r + 1, K), 

- G = S0(4r + 2, C) with I = 0[(2r + 1, C), 

• Type Ee 

- G = E^e-) with [ so(5, 5) © R, 

- G = Eg{C) with [ = so(10,C)®C. 

The parabolic subgroup P corresponds to a closed subsystem i? of A which contains 
all positive roots. In all cases above there exists no element w in the Weyl group of A 
with wR = —R. (Note that this can only occur for a root system of type An, D2n+i 
and Eg.) Hence the classical Knapp-Stein theory does not yield intertwiners — > 
I{—iy). Instead we employ the method of the "spectrum generating operator" by 
Branson-Olafsson-0rsted [2]. Initially this method was invented to determine the 
eigenvalues of a given intertwining operator. However, it also provides a technique 
to construct intertwiners purely in terms of their eigenvalues. We use this technique 
to find non-standard intertwining operators for all unitarizable subrepresentations 
of and answer questions (1), (2) and (3) (see Theorem 13.21 for the reducibility 
question, Theorem l3.3l for the composition series and Theorems 14.51 and 14 . 1 T] for 
the unitarity question). It is noteworthy that non-trivial unitarizable constituents 
other than the unitary principal series (i.e. ly G i{ap)*) only occur for the restricted 
root system A of type D or Eg- Here in particular the study of the unitarity 
question is delicate. To successfully apply the method of the "spectrum generating 
operator" we need to determine whether certain coefficients in the expansion of the 
product of two spherical functions on X vanish (see Lemma 142]) . This is done using 
a result by Vretare [33] and a combinatorial calculation carried out in Appendix El 
We further study the associated varieties of the unitary constituents and find 
some representations with associated variety equal to the smallest nilpotent Kc- 
orbit inpl for the groups SL(l-Ks,IK) (K € {R,C,IH}), S0(2r -M, 2r -M), S0{4r + 
2, C), i?6(-26) ! -E-ece) ^.nd Eq{C) (see Section[5]). These representations are all spheri- 
cal, have one-dimensional asymptotic JiT-support and their Gelfand-Kirillov dimen- 
sion attains its minimum among all infinite-dimensional unitary representations. 
X^-models of these small representations in the spirit of Sahi [37] , Dvorsky-Sahi [5] 
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[7], Kobayashi-0rsted [25], Barchini-Sepanski-Zierau |T| and Hilgert-Kobayashi- 
M. [14] will be the subject of a subsequent paper. 

For the special linear groups G = SL(r + s,K), K e {K, C,H}, with maxi- 
mal parabolic subgroups P = S{GL{r,K) x GL(s,]K)) x M'"^* our results on re- 
ducibility, composition series and unitarity are not new, but were earlier obtained 
by Howe-Lee [15] and Lee [27] (see also jS] [Ml [41] for the cases r = 1). For 
G = S0o(2r + l,2r + 1) Johnson [TO] found the points of reducibility and the 
composition series, but did not answer the unitarity question. All remaining cases, 
in particular the exceptional groups, seem to not have been treated yet. Further 
work on degenerate principal series associated to maximal parabolic subgroups with 
possibly non-abelian nihadical can be found in [3] [9] fTT I [T6 | llT l [2T 1 128 ] [29] . 

It is also worth mentioning the recent work of Clerc [4] and Olafsson-Pasquale [34] 
on Knapp-Stein intertwining operators between degenerate principal series on sym- 
metric i?-spaces corresponding to different parabolics. 

Acknowledgement: For helpful discussions we would like to thank J. Vargas 
and in particular B. 0rsted for bringing the method of the "spectrum generating 
operator" to our attention. 



1.1. Symmetric i?-spaces and root data. Let G be a simple real non-compact 
Lie group with maximal parabolic subgroup P C G whose nihadical is abelian. 
Then, by definition X := G/P is an irreducible symmetric i?-space, cf. [32] [40]. In 
more detail, let P = LN denote the Levi decomposition of P, and let K C G be 
a maximal compact subgroup such that M := L n K is maximal compact in L. 
Since G = KP, we have a natural identification X = K/M. Let q, I and n be the 
Lie algebras corresponding to G, L and N, and let denote the Cartan involution 
corresponding to K C G, resp. t Q Q. Then, there is a (unique) grading element 
Zq G 3([) in the center of [ satisfying OZq = —Zq and such that g decomposes under 
the adjoint action of Zq into 



with eigenvalues —1, 0, 1. Here [ © n is the Lie algebra of P. We note that n = On. 
The involution a :~ Adexp(7riZo) on gc leaves g invariant and acts on n © n by 
— 1, and on [ by 1. This induces a non-trivial involution (also denoted by a) on G 
satisfying a{K) — K such that {K,M,a) is a compact symmetric pair. Therefore, 
X is a compact Riemannian symmetric space. We further remark that n naturally 
carries the structure of a simple real Jordan triple system (see [3T]). In fact, this 
establishes a one-to-one correspondence between irreducible symmetric i?-spaces 
and simple real Jordan triple systems. However, we will not use the Jordan triple 
structure in this paper. 

The Cartan decomposition of g with respect to 9 is denoted by 



1. Preliminaries 



(1.1) 



g = n© [©n 



(1.2) 



g = « ©p. 



Since 9Z, 



V 



Zq, the decompositions p. II) and p.2p are compatible, so 



g = m © nt © Ip © rip , 
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where m = [ n t is the Lie algebra of A/ C L, [p := ( n p, and 

nt -.^ (n® n) nt ^ {X + ex \ X e n} , 
(13) \ / I I J I 

np :=(n©n)np = {X-6iX|XGn}. 

Let a C p be a maximal abelian subalgebra containing Zq, and let A — A(g, a) be 
the corresponding (restricted) root system. Then A splits into A = A_i U AqU Ai, 
where A^ = {A S A | X{Zo) = k}. We choose a positive system A"*" C A such that 
Ai C A+ C AqU Ai, and let . . . , /J^ be a maximal system of strongly orthogonal 
roots in Ai such that /3i is the highest root in A. The number r is called the split 
rank of the symmetric i?-space X. For each /3k, we fix an s[2-triple {Ek, hk, E-k), 
i.e. 

(1.4) [hk,Ek] ^2Ek, [hk,E^k] = -2Ek, [Ek,E^k] = hk, 
satisfying Ek G Qjs^, E^k = -OEk G Q-js^, hk € a. Set Hk := Ek - E^k £ n{. Then 

r 

(1.5) t:=0Mi/fc Cnt 

fe=i 

is by |24l Proposition 4.3] maximal abelian in rif and if 7^ e is defined by 
jj{iHk) '■= 2Sjk, the (restricted) root system A(gc,tc) is given by either 

' ±7j ± 7fe 



l<J,fc<r^\{0} (type a), 



(1.6) , 2 
called the 'unital' case, or 

(1.7) {^^,±1 

called the 'non-unital' case, see [23] Proposition 2.2]. The term 'unital' corresponds 
to the fact, that the natural Jordan triple structure on n comes from a unital Jordan 
algebra precisely if A(0C;tc) is of type Cr- As noted in the introduction, here we 
are interested in the non-unital case (see Tables [T] and [5] for a classification). 



l<j,fc<r \{0} (typeiSa), 








[ 


n 


A.l 


sl{r + 


s,M), s > r > 1 


sl(r,R) ©s[(s,R) eR 


M{r X s,R) 


A.2 


sl(r + 


s,C), s > r > 1 


sl(r,C)©s[(s,C)©C 


M(r X s,C) 


A.3 


sl{r + 


5,11), s > r > 1 


s[(r, H)esl(s,H)©R 


M(r X s,H) 


A.4 




e6(-26) 


so(l,9)®M 


M(l X 2,0) 


D.l 


so(2r -f 


- l,2r+ 1), r > 1 


s[(2r + 1,R) ®R 


Skew(2r + 1,R) 


D.2 


so (4? 


- + 2,C), r > 1 


sl(2r + l,C)eC 


Skew(2r + 1,C) 


E.l 




^6(6) 


so(5,5)eM 


M{1 X 2,0,) 


E.2 




e6(C) 


so(10,C)©C 


M{1 X 2, Oc) 



Table 1. Non-unital irreducible symmetric i?-spaces: non- 
compact Lie algebras 



From the classification we see that the root system A = A(0, a) is either of type 
An, D2n+i or Eq. Notc that in these cases there is no element in the Weyl group 
of A which acts on a by —1. 
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A.l 

A.2 
A.3 
A.4 



D.l 
D.2 



E.l 
E.2 



so{r + s) 
su(r + s) 
sp(r + s) 

u 



so(2r 



■f l)eso(2r- 
so(4r + 2) 



1) 



sp(4) 

66 



so(r) ©so(s) 
s(u(r) ©u(s)) 
sp(r) ©sp(s) 
so(9) 



so(2r 
u(2r- 



-1) 
1) 



sp(2) ®sp(2) 
so(10)@R 



Table 2. Non-unital irreducible symmetric i?-spaces: 
Lie algebras 



compact 



The intersection of the root spaces of A{gc, tc) with Q 9c yields a root system 
■^(^Cjtc) Q tj' non-miital case the multiplicities 

e := dim(tc)±7, , d := dim{tc) ±ij±ik , 6 := dim(tc) + 2i 

2 2 

are independent of the choice of 1 < j < fc < r and the signs (see |301 § 11]) and we 
always have 6 7^ 0. Therefore, the non-unital case splits into two possibilities: 



A(6c,tc) is of type ■ 



Br if 6 7^ 0, e = 0, 
BCr if 6 7^ 0, e 7^ 0. 

For later purpose, it is convenient to introduce the genus p of X defined by 

(1.8) (e + l) + (r-l)d+^. 

Let K denote the Killing form of g and its bilinear complexification. Then 
n{Hk,H,)^ J2 A(i/fe)A(i/,)dim(0c)A. 

AeA(gc,ic) 

We thus obtain 

where e = dim(0c)±7j, d = dim(0c)(±7j±7fc)/2, b = dim(0c)±7j/2, independent of 
the choice of 1 < j < fc < r and signs. It is known that e ^ e + 1, d ~ 2d and 
h ~ 2b, and hence we note that 

(1.9) p = -^KiHi,He), 

independent of 1 < ^ < r. 

We fix a lexicographic ordering of roots in A(€c, tc) by setting 71 > 72 > • • • > 7r. 
The corresponding set of positive roots is denoted by A+(tc, tc). A straightforward 
calculation shows that the half sum of positive roots is given by 



(1.10) p' = J2pklk with p, :=(r-fc)| + § + | = H2^-(/c-l)| 



'2- 

k=l 



For the discussion of induced representations we set ap := M.Z0 and Ap := 
exp(ap) and let P = MpApN be the corresponding Langlands decomposition of 
P. Let 7 G ap be the linear functional defined by 7(^0) = 1- Since ()1.1[) is 
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the decomposition of g corresponding to the adjoint action of ap, it follows that 
A(g,ap) = {±7}- The parabolic P C G corresponds to the choice of 7 as the 
positive root, and hence 

n 

(1-11) P^^l 

is the half sum of positive roots where n = dimn. 





n 


n 


P 


r 


d 


e 


b 


A.l 


M(l X s,R), s > 1 


s 


{s + l)/2 


1 








s — 


1 


A.2 


Af(l X .s,C), s > 1 


2s 


s + 1 


1 





1 


2(s- 


1) 


A.3 


M(l X s,H), s > 1 


As 


2(s + l) 


1 





3 


4(s- 


1) 


A.l 


M{r X s,R), s > r > 1 


rs 


(r + s)/2 


r 


1 





s — 


r 


A.2 


M{r X s,C), s > r > 1 


2rs 


r + s 


r 


2 


1 


2(,s- 


r) 


A.3 


M{r X s,M), s > r > 1 


4rs 


2{r + s) 


r 


4 


3 


4(s- 


r) 


A.4 


M{1 X 2,0) 


16 


12 


1 





7 


8 




D.l 


Skew(2r + r > 1 


r(2r + 1) 


2r 


r 


2 





2 




D.2 


Skew(2r + 1,C), r > 1 


2r(2r + 1) 


4r 


r 


4 


1 


4 




E.l 


Af(l X 2,Os) 


16 


6 


2 


3 





4 




E.2 


M(l X 2,Oc) 


32 


12 


2 


6 


1 


8 





Table 3. Non-unital irreducible symmetric i?-spaces: structure constants 



1.2. K-types. In this section, we describe the decomposition of L'^{X) into K- 
irreducible subspaces by using the Cartan-Helgason theorem. For this we assume 
that G has trivial center. Since K is not necessarily simply-connected and M is 
not necessarily connected we have to be careful in lifting Lie algebraic results to 
the group level. 

For m = (toi, • . • , mr) E II let m > denote the condition 

mi > m2 > • • ■ > > 0. 

The main goal of this section is the following. 

Proposition 1.1. Let X be a non-unital symmetric R-space. Then the space 
LP'{X)k of K -finite vectors in LF'{X) decomposes into 

m>0 

where for m > the subrepresentation V"^ is the irreducible unitary M -spherical 
K -representation with highest weight Am X)i=i™i7i- each the space of 
M -spherical vectors is one- dimensional and there is a unique M-spherical vector 
e V"" with 4>^{xo) ^ 1. 

We first apply the Cartan-Helgason theorem on the Lie algebra level. 

Lemma 1.2. The highest weight of an irreducible ^-representation with an m- 
spherical vector vanishes on the orthogonal complement ofi in any maximal torus of 
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t containing t. The possible weights which give unitary irreducible m-representations 
are precisely given by 

^+({-) _ I {1^1=1 \ ti e ^Z,t, - t^ e Z,ti > . . . > tr > 0} in case B, 
™ I {X]i=i I G ^1 ^ • ■ • ^ ^i! 0} in case BC . 

Further, in each irreducible m-spherical t-representation the space of m-spherical 
vectors is one- dimensional. 

Proof. By the Cartan-Helgason theorem [131 V§4, Theorem 4.1] the set A+(£) of 
highest weights of m-spherical irreducible {-representations consists of all A G 
such that 

4^eNo forallae A+(«c,ic), 
K[a,aj 

where we identify = tc using k. Writing A = ^j7j with tj € R we find 



K{a, a) 




, .7 < k, 



which implies the claimed statement for A+(4). Finally, [131 VI Lemma 3.6] states 
that the space of m-spherical vectors is one-dimensional. □ 

In order to lift this result to the group level, we need the following lemma. 

Lemma 1.3. Let T = cxp(t) C K. 

(1) The exponential map t — > T has kernel 27rZ_ffj . 

(2) T meets every connected component of M , i.e. M = {T D M)Mq. Moreover 
(1.12) TnM ^ exp nZHj 

Proof. Let H := ^ ^ with tj G R, and consider the adjoint action of 

exp(i?) on gc- Since G is the adjoint group of g, the element H is in the kernel of 
exp if and only if Ad(exp(iJ)) = id. Since 

Ad(exp(i7))X = e^J=i *^^^"^^X for X € {gc)x, 

and ^k{Hi) = — 2i(5fcf, the decomposition ()1.7p immediately yields statement (1). 
For (2), we first note that K = MqTMq and therefore M = (T n M)Mo, see [HI 
Chapter V, Theorem 6.7]. Let H Y.]^itjHj with tj e R. Since M = Zxiap) 
we have to show that Ad(cxp(iJ))Zo ~ Zq if and only if tj € ttZ for all j = 1, . . . , r. 
Set := + E-j with E±j defined in ([T4l) . Since ad(iJj)Zo = -Fj and 

ad(i/fc)(F,) = 2,5,fe/ifc, 
ad(Ffe)(/i,) = -2(5,feFfc, 

we find that 



Ad(exp(iJ))Zo = exp(ad(H))Zo = - - ^ ((l - cos(2tj)) /ij + sin(2tj) F 



and the claim follows. □ 
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Proof of Proposition \l.l\ First note that by Lemma [1.31 (1) the lattice 
consists of analyticaUy integral functionals for K . Hence, for m > the t-action 
on V™ integrates to an action of K. Every m-spherical vector (j)-^ E V™ is then 
automatically Aio-spherical. We claim that an Aio-spherical vector (f)^ is also M- 
spherical. In fact, by Lemma 11.31 (2) it suffices to check that k ■ 0rn = 4>m for 
keTCiM. Write 



(fco • Vm) dfco 

I Mo 

for some highest weight vector Vm G V"^. Then by (|1.12p we have fc • = Vm and 
hence we obtain 

fc • 0m = / (fcfco • Vm) dfco = / (fcofc ' fm) dfco 
J Mo -J Mo 

(fco • Vm) dfco = 0m. 



'Mo 

It remains to show that the representations V"^ comprise all irreducible iC-represen- 
tations with an M-spherical vector. By Lemma 11.21 it suffices to show that a 
_ftr-representation V with highest weight X]j'=i'^j7j' '^i ^ ■ ■ ■ ^ n^r > 0, can 
only have an Af -invariant vector if rrij g Z for all j = 1, . . . ,r. Let 7^ be 
an Af-invariant vector and let v be a highest weight vector. If ( • | • ) denotes the 
if-invariant inner product on V then {k ■ v\(j>) = {v\<f>) ^ for all fc G A/. By 
Lemma ri. 31 (2) the element fc := exp(7riJj) is contained in A/ and k ■ v = e'^™'"-iv. 
Hence rrij e Z for all j = 1, . . . , ?■ and the proof is complete. □ 



2. Degenerate principal series and the spectrum generating operator 
For the rest of this article let X be a non-unital symmetric i?-space. 

2.1. Degenerate principal series. Recall the element Zq e ap from p.ip . We 
identify (ap)^ with C by the map v h-s- ^v{Zq). Then p from (|l.lip corresponds to 
^. (The reason for this normalization is the simplicity of the formulas in Propo- 
sition 12.31 ) We form the induced representation (normalized smooth parabolic 
induction) 

I{v) Ind^(l ®e''®l) = {ipe C^iG) : ipigman) = a-^-Pip^g) 

y g e G,man e MpApN}. 

Denote by tt^, the corresponding action of G on by left-translation. 

Since G = KP, restriction to K defines an isomorphism 1(1/) — > G°°(X). Denote 
by (pi/, G°°{X)) the corresponding representation of G, i.e. 

Pu{9){'f\K) := {Tr,.{g)'f)\K geG,Lpe I{v). 

The restriction of to K is the left-regular representation on C°°{X) and hence, 
by Proposition ll.il the A'-finite vectors G°"{X)k decompose into 

G^{X)k^^V'^, 

m>0 

each i^-type V"^ appearing with multiplicity one. In what follows we will identify 
with the corresponding subspace in G^{X) or I{v) and it will be clear from 
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the context which identification we use. Further note that 

/ ^p{k)i}}{k)dk 

J K 

provides a sesquihnear form on C°°{X) which is invariant under the representations 
Pv ® P-v, i-e. 

{pv{g)^,p~v{9)'>P)mx) = {v,i'')L^x), ¥','0 € c°°{x). 

2.2. The spectrum generating operator. Let us recall some results from [2]. 
Denote by dpi, the differentiated representation of g on C°°{X). Let (•, •) := 
• , • ) be the renormalization of the Killing form of g such that {Zq, Zq) = 1. 
On i the bilinear form — ( ■ , ■ ) is positive definite and we choose an orthonormal 
basis {Ta)a C Tit of the form = + ONa, Na e n. Consider the quadratic 
element 

a 

Then by [2] Lemma 2.2] the right regular action TZ-p of V leaves C°°{X) invariant. 
This operator is called the "spectrum generating operator". In our case we can 
obviously express V in terms of the Casimir elements C{ e and Cm € U{m) of 
t and m. respectively, taken with respect to the inner product ( • , • ) (cf. [2l Remark 
2.4]): 

Since the right regular action of Cm on C°°{X) is trivial and the right regular 
action of C{ on C°°{X) coincides with the left-regular action we find that the 
spectrum generating operator TZ-p agrees with the Laplacian dp,y{Ct) of X which 
is independent of i'. As Casimir operator. dpj,(C{) acts on each K-type V"^ by the 
eigenvalue 

T^m ■= - (Am + Sp', Am), 

where we identify 5^ ic via the non-degenerate bilinear form (•,•). A short 
calculation using (|1.9p shows that we have (7^,7^) — ~^Sjk and hence 

r r 

(2.1) n^^^Y, (™j- + 2p,)m, = - ^ (m^ + {p - 1 - {j - l)d)m,). 

Now define a function uj : gc ^ C°°{X) by 

uj{T){k) (T,Ad(fc)Zo). 

Since Zq G pc and pc is Ad(ii')-invariant, we clearly have wjtj, = 0. By the 
invariance of the bilinear form ( • , • ) we further obtain that tu is iiT-equi variant. 
Let m{uj{T)) : C°"{X) C°°(X) be the multiplication operator ip i-> uj{T)ip. 
Then by [H equation (2.1)] we have 

{dp, - &p,>){T) = ^ v')m{Lo{T)), TG0C. 

For m > let Pm : L'^i^) ^ denote the orthogonal projection onto V"^ C 
L'^{X). Then by O Corollary 2.6] we further have for m, n > 0: 

(2.2) 2P„o dp,(r)|y™ = (^:/ + 7r„-7rm)Pnom(w(T))|v™, T e Pc- 
Consider the map 

$m : Pc ^ C°°{X), T®^^ m[u{T))^ . 
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One readily checks that <i>in is /C-equivariant, and hence there is a subset Q 
{m e Z*^ I m > 0} such that 

To determine exphcitly we need the following lemma on the spherical vectors 
0m € V"^, m > 0. For convenience we set (pm := for m e Z'' not satisfying 
m > 0. 

Lemma 2.1. Let X be non-unital and m > 0. Then 

r r 

where 



k=l k=l 



A{m.,k) 



p (2(TOfc + pfc) + I + l)(2(mfc + Pfc) + I + e) 
2n 2{mk + Pk){2{mk + Pk) + I) 

n 



X 



B{m,k) 



ijmk + Pk) - {m-j + Pj) + f + pk) + (mj + pj) + |) 

j^k ^^^'^ + ~ ^'^1 + PMi^k + Pk) + [rrij + pj)) 
p (2(mfc + pfc) - I - l)(2(mfc + p^) - | - e) 
2n 2(mfc+pfc)(2(mfc+pfe)-l) 

((wfc + Pfe) + + Pj) - f )((mfc + Pk) - (mj + pj) - |) 



X 



n 



((™fc + Pfe) + (™j + Pj)){{'mk + Pfc) - (mj + pj)) 



C(m) = l- ^ A(m,fc)- ^ B(m,/c) 



/c=l A:=l 
m+ej^->0 m— efc>0 



as rational functions mm. /n particular, 



A(m, fc) ^ /or m + efc > 0, • A(m, fc) = /or m + ^ 0, 

• i3(m, fc) 7^ /or m — > or • i3(m, fc) = /or m— ^ wif/i 

fc = r, TTir = 0, pr = 2 ' either k ^ r or k = r^ p.,- ^ \ 

Proof. Note that pc is an irreducible 4-module and the element Zq E pc is m- 
invariant. Moreover, 71 is the highest weight of pc, so pc = V'^'^. Now a; : pc — ?> 
C°°(X) is t-equivariant and therefore uj{Zo) is a scalar multiple of the spherical 
vector (pe-^ ■ Since (eM) = 1 = a;(Zo)(eAf ) we even have Ld{Zo) = (pe^ . Using 
Theorem 4.8] we obtain that the claimed expansion of the product 4>ei4>m holds 
with 

A, , ^ ^ c(-i(7i+p))cH5+(m + p)) 



c(-ip)cH(5+(m + p) + 7i)) 

S(m,fc) = 



c(-j(7i + p))c{-iSk (m + p)) 



c{-ip)c{-i{S^^ (m + p) + 71)) ' 
where c(A) denotes the c-function of the root system S = A(€c,ic) and 

'^fe^('^l' ■ • • ' ^r) — (i-^fci ^2, . . . , Afe_i, Ai, \k+l, • • • , Ar). 
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By the Gindikin-Karpelevich formula 



'^^^ '° alio + 1 + ^^mUl^^c. + + ^^)) 

with some constant cg 7^ and Sg •= {ct G S+ : ^ ^ E}. Since S is of type B or 
BC we have 

So = {? I 1 < J < r} U I 1 < J < A < r} 

with multiplicities 

2 2 2 

Using the identity r(z)r(z + i) = y/1^2^-^'^r{2z) for the gamma function we obtain 
the following general expression for the c- function at A = X]j=i ^jlj- 



Lr(i(| + l + 2zA,))r(l(|+e + 2zA,)) 

- Afe))r(z(A, + A,)) 



X 



n 



i<,<fe<. r(| + *(A, - A,))r(f + ^(A, + A,)) 
with some constant Cq 7^ 0. With (jl.lOp we find 

c{-ip) 

(2pi)(2pi + l) -j-j- 



and 



(2pi + I + l)(2pi + I + e) -^Jf (pi - + Dip, + pj + f ) 

(2pi + 1) ^ ^ 
2r(2pi + 1 + 1) 2n 



c(-z^,^(m + p)) 
cH(5±(m + p) + 7i)) 

(±2(77ife + p^) + I + l)(±2(j7ifc + pfe) + I + e) 



X 



n 



±2(mfc + pfc)(±2(mfe + pk) + 1) 

(±(mA.. + pfc) - (mj + pj) + f )(±(mfc + pfc) + {rUj + pj) + f ) 



(±(mfc + Pfc) - [nij + pj))(±(mfc + pk) + (mj + p^)) 



and thus the formulas above follow. We now prove the remaining claims for i?(m, A;), 
the corresponding results for A{m, k) are shown similarly. First note that we can 
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rewrite B{m, k) as follows: 

p (2(?7ifc + pfc) - I - l)(2(mfc + pk) - I - e) 
2n 2(TOfc + pfe)(2(mfc + /9fe) - 1) 

{mj + pj) - (mfe + Pk) + i -TT (»7ife + Pk) - (mj + Pj) 



('"j + ^ ("^fe + ^fe) 7>i ('"'^^ + Pk) - {mj + Pj) 



n 



^ {mk+Pk) + {mj+Pj, 2 
{mk + Pk) + [riij + Pj) 



Recall from p:TU)l that = (r - j)| + f + | > | > i > for every j = 1, . . . , r. 
Further, in the case r > 1 we have — = — j)| > | > for j < ^. Therefore, 
for m > the denominator of i3(m, k) is always non-zero if 2(mfc + p^) — 1 7^ 0. 
Since G No we can only have 2{rnk + Pk) — 1 = for = 0, pfe = ^. Further, 
since Pr = | + | > g; and Pfc > | + | > | for 7^ r we can only have pk ^ \ 
for fc = r and & = 2,e = 0. In this case, i.e. k = r^m,. = Q,pr ~ 5, the factor 
2{mr + p, ) — 5 — e in the numerator cancels with the factor 2 (to,, + p^) — 1 in the 
denominator. It is further easy to see that all other factors in the numerator are 
non-zero, so in this case B{m., r) ^ 0. If we suppose that m, m — ek > 0, i.e. 

TOl > . . . > TOfe > TOfe — 1 > rUk+l > • • • > TOr > 0, 

using the convention rrir+i = 0, then all terms in the numerator and denominator 
are non-zero and hence again i?(m, k) ^ 0. Finally, suppose that m — Cfe ^0, i.e. 
TOfe = nik+i- If fc 7^ r then the factor (to^, + Pfc) — [irij + Pj) — ^ hi the numerator 
vanishes for j = k + 1. If fc = r, p^ 7^ ^ then to^ = and the factor 2 (to^. + Pfe ) — | — e 
in the numerator vanishes. This completes the proof. □ 

Corollary 2.2. For each m>Q we have 

{m ± > 1 1 < j < r} C 5*^ C {m, m ± > 1 1 < j < r} . 
Further, m G Sm if and only if C(m) ^ 0. 

Proof. For every n £ Sm the i^-module appears in the tensor product pc^V™'. 
By [IHl Proposition 2.1] the highest weight of a subrepresentation of pc <E) V"^ is 
the sum of the highest weight of V"^ and a weight of pc- Since the restricted 
weights of Pc are contained in the set {il^li \l<j,k<r}U 1 1 < j < "r} 

the only possible spherical representations contained in pc ® V"^ satisfy by 
Proposition 1 1 . 1 1 either n = m or n = m ± for some 1 < j < r. Hence the second 
inclusion follows. The first inclusion is an immediate consequence of Lemma 12. II 
Now if C(m) 7^ then clearly m G 5m- It remains to show that C(m) = 
implies m ^ 5m- For this we first note that by |42| Proposition 2.3] the K- 
representation pc is weight-multiplicity-free (with respect to any Cartan subalgebra 
of t). Hence, again by |181 Proposition 2.1], the representation V"^ has multiplicity 
at most one in pc (8) V"^. This implies that Honix(pc (E) V"^, V"^) is at most one- 
dimensional. Therefore, by [201 Proposition 3.1] every non-zero i^T-homomorphism 
pc ® V"^ — !• V"^ maps the spherical vector Zq ® 0m to a non-zero scalar multiple of 
the spherical vector 0m € V"^. Since C(m) = the liomomorphism F : pc^V"^ ^• 
y™, T(g>ip H> Pm("i(a;(T))(p) maps Zo^cf)^ to zero and therefore i^(pc(8)F™) = 0. 
Hence m ^ 5m and the proof is complete. □ 
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Proposition 2.3. For m> we have 

More precisely, we have for T G pc 

2P„+,^ o dp,iT)\v^ = f(iy + m+0-))Pm+e, o m{Lu{T))\v^, 

2P„_,^ o dp,{T)\vr. = f{iy - m-(j))P™_e, o TO(cj(r))|y™, 

2Pm o dp,{T)\v^ ^ fiyP^ o m{Lo{T))\v^, 

where 

m+(j) nij + - - ( j - 1)-, roT {j) := {rrij - ^) + ^ - U ^ 1) 2" 

Proof. The first claim follows directly from Corollarv l2.2l For the explicit formulas 
we use (|2.ip and (|2.2p and observe that 

2n f p d 

(2-3) TTm+e, - T^m ^ — [rrij + - - (j - 1)- 

This completes the proof. □ 

3. ReDUCIBILITY AND COMPOSITION SERIES 

We now determine all points of reducibility for and find the complete com- 
position series in these cases. 

3.1. Reducibility. Recall the constants A{m., j), B{m.,j), C(m) from Lemma 
and m+(j), m^{j) from Proposition 12.31 

Corollary 3.1. For every m > and s GR we have 
(3.1) dp,{Zo)(t>m = — ^ (^(m, k){iy + m+(fc))0„+e, 

\ 2n 

+ B(m, - m"(fc))(?!)m_eJ H C(m)i^^in- 

/ p 

Proof. This follows directly from Lemma 12.11 and Proposition 12.31 □ 

Theorem 3.2. Let X be a non-unital symmetric R-space. Then the principal series 
representation I{v) is reducible if and only if either 

^, P , ^d P , sd 

i.€-No-| + (j-l)- or ^eNo + |-(.?-l)- 

for some 1 < j < r. In particular, is irreducible for all v G iM. 

Proof. By Corollarv l3.1l it follows that the representation liy) is irreducible if 

^ -No - I + (j - 1) ^ and ^ ^ No + I - (j - 1 ) ^ , 

because then every if -type can be reached from every other if-type by 
successive application of dp^. Now suppose that = — m — | -t- (j — 1)^ for some 
TO G No and 1 < j < r (the other case is handled similarly). Then by Proposition 
we have 

i'm+e, O dp,(T)|y™ = 0, T G 0c 
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for all m > with mj = m. Hence, the proper subspace of consistmg of 
all A'-type V"^ with m > 0, irij < m, is g-stable and therefore /(i/) is reducible. 
Finally note that, since 

p , ,d, de+lbb 

--(?■- 1- = (r-?)- H h->->0 Vl<7<r, 

the two possibilities cannot occur simultaneously. □ 



3.2. Composition series. Using the observations from the proof of Theorem 13.21 
it is easy to determine the composition series in the case where /(z^) is reducible. 
For this we let 

rrij <~v~ - + [j - 1)- 



£j{v) <^ m > 



mj>u-- + {j- 1)- 



rj{v) := |m > 
for 1 < J < r and define the following subspaces 

Then we have the inclusions 

(3.2) {0}CLi(i.)C...Ci,(zy)C/(j/)^, 

(3.3) /(i^)if C i?i(j/) C ••• C i?^(i/) C {0}. 
The following result is immediate 

Theorem 3.3. Let X be a non-unital symmetric R-space and assume I{v) is re- 
ducible. 

(1) If V ^ -No - f + (j - l)f let I < ii < ■ ■ ■ < it < r be such that v S 
—No — — l)f */ o,nd only if j G {ji, . . . , Jt}. Then the composition series 
of I{v)k is given by 

{Q}QL,Ay)Q---QL,,{v)QI{y)K. 

(2) Ifv e No + §-(j-l)f letl<ii<---< it <rbe such thatv No + §-(j-l)f 
if and only if j G {ji, . . . ,jt}. Then the composition series of I[v)k is given 
by 

4. Unitarity 

Let W C I{y) be a subrepresentation. Then W is unitary if and only if there 
exists an intertwining operator T : W ^ with strictly positive eigenvalues 

on the -fC-types occurring in W. In fact, in this case the invariant inner product on 
W is given by 

{v,w)w ■■= {v,Tw)l^x) = ^ ^(m)(wm,u'm)L2(x), v,weW, 

where W^(m) denotes the A'-isotypic component of V"^ in W, Vm € W{m.) the 
orthogonal projection (with respect to the L^-inner product) v G W onto W{ni), 
and t{m.) the eigenvalue of T on W{m.). To find such intertwining operators and 
their eigenvalues we employ the method of the "spectrum generating operator" by 
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Bransoii-Olafsson-0rsted |2] . For this recall the eigenvalues TTm of the spectrum 
generating operator TZ-p from Section [2] 

Theorem 4.1 (j2j Theorem 2.7]). Let W C I[v) he Q-invariant. Then a map 
T : W ^ li—v) with T|ym = t(m)idv'm is an intertwining operator if and only if 
for any m, n > with VF(m), VF(n) ^ and n e Sm we have 

(4.1) (7r„ + fiy)t{n) = (7r„ - tt^ - fv)tim.). 

This immediately gives a necessary criterion on the coefficients C(m) in p.ip : 

Lemma 4.2. Let W C I^i/) be a unitary subrepresentation of Then either 

v€im. or C(m) for all m > with W{m) ^ 0. 

Proof. Let T : W ^ ^(^f^) be an intertwining operator with strictly positive eigen- 
values t(m) on VF(m) 7^ 0. Suppose there exists m > with W{m.) 7^ and 
C(m) ^ 0. Then by Corollary [221 we find that n := m G Sm and in this case (|4.ip 
implies 

{iy + V)t{m) ^ 

which is only possible for i/ E iM. since t{m.) > 0. □ 

Note that if C(m) = then by Corollarv l2.2l we have m ^ Sm and the obstruction 
in the proof of Lemma 14.21 does not occur. In this case (|4.ip gives recurrence 
relations for the eigenvalues t(m) which determine T uniquely up to scalar multiples 
on every irreducible subrepresentation. 

Thus we need to find out for which m the constants C (m) vanish. If X is unital 
then P is conjugate to its opposite parabolic and hence there exist Knapp-Stein 
intertwiners — > I{—v) for infinitely many values £ R. Therefore C(m) = 
in these cases. For non-unital X, however, it can very well happen that C(m) 7^ 0. 
From the formula for C(m) given in Lemma [2.11 it is hard to determine for which 
m we have C(m) = 0. We use the following result which is proved in Appendix O 

Proposition 4.3. Assume 2mr + 2pr > 1. 

(1) For d 0,2 we have 

rb pb{e+l-l) ( ^ {2mk + 2p,r-{d^l)^ \ 
^""^ 2n 2nd{d-2) \ ii (2mfc + 2pfc)2 - 1 j' 

(2) For d e {0, 2} we have 

rb , p6(e+|-l) ^ 1 
C(m) = > -r. 

^ ' 2n 2n ^ 1 " (2™fc + 2pfc)^ 

We now study the constants C(m) separately for all non-unital symmetric R- 
spaces and determine all unitarizable subrepresentations using Theorem 14. II 

4.1. The cases g = sl{r + s, K) and g ~ ^6(-26)- Let g = sl{r+s, K) with parabolic 
corresponding to [ = s{g^{r, K)®qI{s, K)) for K G {M, C, H} and s > r > 1 or K = O 
and r = 1, s = 2. Here s[(3,0) = e6(_26)- 

Lemma 4.4. For q ~ sl(r + s, K) with parabolic corresponding to I ~ s(g[(r, K) © 
0[(s,IK)) for K € {R.CR} and s > r > 1 or K = O and r = 1, s = 2 we have 
C(m) = if and only if m = 0. 
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Proof. For g = sl{r + s, R) and g = s[(r + s, C) this follows from |151 Propositions 
2.5.7 & 3.5.3] and for g = sl{r + s,m) from |23 Proposition 3.14.2]. It remains to 
consider the rank 1 case q ~ ^e(-2e)- Here n = 16, p = 12, r = l, d = 0, e = 7 and 
6 = 8 and with Lemma 12.11 or Proposition 14.31 (2) we find after a short calculation 

CM- -("^ + 11) 



4(TO + 5)(m + 6) 

which shows the claim for this last case. □ 

Theorem 4.5. Let g = sl(r + s, K) with parabolic corresponding to I ~ s(gl(r, K) © 
0[(s, K)) for K e {R^CM} and s > r > 1 or K = O and r = 1, s = 2. Then I{v) is 
irreducible and unitary if and only if v ^ iM. // I[v) is reducible, the only unitary 
subrepresentation resp. subquotient that can occur is the trivial representation and 
it occurs as subrepresentation resp. subquotient for = — f ''^^■^P- — §• 

Proof. With Lemma 14.21 and Lemma 14.41 we find that either v G iM. or the sub- 
representation resp. subquotient is isomorphic to the trivial representation. 
That the trivial representation occurs as subrepresentation for = — f foUo'^^s 
from Theorem 13.31 □ 



Remark 4.6. For g = s[(r + s,M) and g = sl{r + s,C) Theorem 14.51 was shown 
by Howe-Lee [HI Propositions 2.5.7 & 3.5.3] and for g = sl{r + s,M) by Lee HZl 
Proposition 3.14.2] (see also EI] for the case r = 1). The case g = te(-26) 
does not seem to have been treated before. 

4.2. The cases g = C6(6) and g = e6(C). Next we study the two exceptional cases 
= e6(6) and g = e6(C) of split rank r = 2. 

Lemma 4.7. Suppose g = e6(6) or g = eelC). Then C(m) ~ if and only if 
TO2 — 0. 

Proof. We use Proposition 14.31 (1) to calculate C(m) explicitly. 

(1) For g — 66(6) we have n = 16, p = 6, ?' = 2, e = 0, 6 = 4 and d — 3, and hence 

, . _ m2(2mi + 3)(2toi + 7)(m2 + 2) 
~ 4(7711 + 2)(mi + 3)(2m2 + l)(2m2 + 3) 

from which the claim directly follows. 

(2) For g = e6(C) we have n = 32, p = 12, r = 2, e = 1, 6 = 8 and d ~ and 
obtain 

_ m2(mi + 3)(mi + 8)(m2 + 5) 
^ 4(mi + 5) (mi + 6)(m2 + 2)(m2 + 3) ' 
This completes the proof. □ 

Recall the definition of the subspaces Lj{v) from Section [3] 

Theorem 4.8. Let either g = e6(6) or q ~ Then L{v) is irreducible and 

unitary if and only if v G iM. The only unitary subrepresentations that occur in a 
reducible L{v) are given as follows: 

(1) Let g = e6(6). 

• The subrepresentation ii(— 3) is the trivial representation and hence uni- 
tary. 
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• The subrepresentation L2(— |) with K-type decomposition and norm given 
by 



,0) 



m=0 



r(m 



m=0 



r(m + f) 



m\\L^(X) 



for v,n G l/^™'*^) is unitary. 
(2) Le<0 = C6(C). 

• T/ie subrepresentation 6) is t/ie trivial representation and hence uni- 
tary. 

• r/ie subrepresentation L2{—3) with K-type decomposition and norm given 
by 



L,{-3) = 



,0) 



m=0 



E 



E 



r(m + 9) 
r(m + 3) ' 



-'■inWL^iX) 



for v,n G l/^™'*^) is unitary. 



Proof. By Lemma [4.21 and Lemma [4.71 we find that a subrepresentation W C /(i/) 
can only be unitary if either e zM or only contains A'-types of the form V^^"''*^), 
m G Nq. 

(1) For g = e6(6) this is by Theorem 13.31 precisely the case if either = —3 or 
= — |. For I' = —3 the subrepresentation Li(— |) is the trivial representation 
and hence unitarizable. For = the subrepresentation i2(~|) contains the 
AT-types l/f"'") foj. 

TO G No and it remains to show that this subrepresentation 
is unitary. By Theorem 14. II we obtain the following recurrence relation for the 



eigenvalues t{m) of an intertwiner T : i2(~f ) 

t{m + 1) ""(m+l.O) ^ ""(m.O) 



2",, o 

2to 



+p 



on y('"^0): 
+ 3 



m 



t(m) 



"■(m+l.O) - ""(m.O) + 



2to + p — 3 



TO 



Choosing ^(to) := r(m + |)r(m + f we obtain an intertwiner with strictly 
positive eigenvalues and hence an invariant inner product on L2(— |). 
(2) Using the same argument for g = e6(C) we find for i/ — —3 the subrepresenta- 
tion La (-3) with AT-types and every operator T : L2(-3) /(3) with 
eigenvalue t{m) on V^"''^^ satisfying 



t(TO+l) T^im+lfi) 



'■(m.O) 



_ 2m 



p + 6 TO + 9 



provides an invariant form on L2{—3). 



2n , 



2m + p — 6 



□ 



4.3. The cases g = so(2r + 1, 2r + 1) and g = so(4r + 2, C). Finally we treat the 
remaining cases g = so(2r + 1, 2r + 1) and g = so(4r + 2, C). 



Lemma 4.9. Suppose that g = so(2r + l,2r + 1) or g = so(4r + 2,C). 
C(m) — if and only if nir — 0. 



T/ie 



Proof. Again we calculate C(m) explicitly using Proposition 14.31 
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(1) For Q = so(2r + 1, 2r + 1) the rank of X is r and n = r(2r + 1), p = 2r, e = 0, 

— 1 = it follows from 



b ~ d ~ 2, pk = 2i__2fe+i^ Since in this case e 



Proposition 



(4.2 



(2) that for to^ > 

rb 
~ 2n 

2pr -■ 



C(m) 



1 



2r + 1 

Now let nir = 0. Since 2mr + 2pr = 1 we cannot directly apply Proposi- 
tion (2). Using the formula in Lemma [TT] we find that 

p 2mr - 1 TT (2mr - 1)^ - (2toj + 2p^)'^ _ 1 



i3(m, r) 



2n 2mr 



In 

j<r 



{2mr + 1)2 - (2TOj + 2pj)2 



2r + 1 



and hence non-zero. In the derivation of the formula in Proposition 14.31 (2) the 
term i?(m, r) was taken into account although m — ^0. Therefore we can 
determine C(m) by adding B{m,r) to (|4.2p and obtain 

C(m) ^ — \- B{m,r) 



, 0. 

2r + l 

(2) For g = so(4r + 2,C) the rank of X is r and n = 2r(2r + 1), p 
b^d^A, pk= 4l^4fc±3 and we find with Proposition US] (1) 



4r, e 



1, 



C(m) 



2r 



n (mj- + 2r - 2,7) (mj + 2r - 2j 3) 
, . (nij +2r- 2j + 1)(toj + 2r - 2j + 2) ' 



For m > all factors in the product are strictly positive except the one for 
j — r which is equal to 

mr{mr + 3) 
[rrir + l){mr + 2) 

Hence the whole product vanishes if and only if = 0. □ 

Remark 4.10. For g = 50(2?- + 1, 2r + 1) Lemma l4?9l was already shown by John- 
son [ini Proposition 3.6 (iii)] using different methods. 

To express the invariant inner product in these cases we define the Gindikin 
Gamma function rfc^d(s) as a meromorphic function of s € C*^ by 

d 
'2 



rM(s) :=nrh.-(j-i) 

We further identify a complex number cr G C with the vector (cr, 



,cr) e 



Theorem 4.11. Let either g = so{2r+l,2r + 1) or g = so(4r -1-2, C). Then I {v) is 
irreducible and unitary if and only if v € iM. The only unitary subrepresentations 
that occur in a reducible liv) are given as follows: 

(1) Let g = so(2r -I- l,2r -I- 1). For j = 0, l,...,r — 1 the subrepresentation 



^j+i 



(— (r — j)) with K-type decomposition and norm given by 



i-ir-j)) 



e 



m>0, mj-^i— 



m>0,mj4.i— 
00 

E 

m>0, rrijj^i—Q 



- j+1,2 



(m+(2r-j)), 



rj+i,2(m + j) 



2 



for Vm G V"^ is unitary. 
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(2) Lei g = so(4r + 2, C). For j = 0, 1, . . . , r— 1 the suhrepresentation Lj+i{—2{r - 
j)) with K-type decomposition and norm given by 



L,+i(-2(r-j)) 



e 



2 

E ^ " 

m>0, mj + i— 

for € is unitary. 



m>0,rnj-|-i— 

CO 

E 

m>0, mj + i— 



r,+i.4(m + 2j) ll^-ll^^m 



Proof. Using again Lemma 14.21 and Lemma 14.91 we find that a subrepresentation 
W C /(i/) can only be unitary if eitlier ly E iR or W only contains ilT-types of the 
form V"^ with nir- = 0. 



(1) Using Theorem l3.3l we find that for g = so(2r + l, 2r + l) the subrepresentations 
of /(i/) containing only V"^ with m,. = occur for ly = — (r— j), j = 0, 1, . . . , r — 
1, and are given by Lj+i{—{r — j)). As in the proof of Theorem 14.81 we use 
Theorem 14. II to find that a map T : Lj+i{—{r — j)) — J- I{{r — j)) which acts by 
a scalar t(m) on the K-type 1/™ is an intertwining operator if and only if 



t(m + efc) 



TOfc 



{2r-k-j + 1) 



t(m) 



2)1 , 



Choosing 



t(m) 



mk + {j-k+l) 
rj+i^2(m+(2r-j)) 



r,+i,2(m + i) 

we obtain an intertwiner T : Lj^i{—{r — j)) — > /((/' — j)) with strictly positive 
eigenvalues and therefore it provides an invariant inner product on (r — 

.?))• 

(2) Using Theorem 13.31 we find that for g = so(4r + 2, C) the subrepresentations of 
/(i/) containing only V"^ with nir ~ occur for ly = — 2(r— j), j = 0, 1, . . . , r—1, 
and are given by Lj+i(— 2(r — j)). The condition ()4.ip for the eigenvalues t(m) 
of an intertwiner T : Lj-^i{—2{r — j)) on the i^T-type V"^ reads 

t(m + efc) _ T^m+Efc - - y^J^ _ mfc + 2(2r - A; - j + 1) 



i(m) TTm+e^ - TTm + y^I^ 'Tlfc + 2(j 

which shows the claim by the same method as in (1). 



fc+1) 



□ 



5. GeLFAND-KIRILLOV dimension AND ASSOCIATED VARIETIES 

We calculate the Gelfand-Kirillov dimension of some unitary representations we 
found in the previous section and use it to find their associated variety. 

5.1. Gelfand-Kirillov dimension. Let us briefly recall the definition of the Gelfand- 
Kirillov dimension. We denote by 

the canonical filtration of the universal enveloping algebra U{g) of g. For a finitely 
generated Z//(g)-module W choose a finite-dimensional generating subspace Wo C W 
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and define the Gelfand-Kirillov dimension of W by 

r^T^ J- /w\ r logdimZife(0)Wo 
GK-dim(W^) := limsup . 

fc->oo log k 

This definition does not depend on the choice of Wq . 

For an irreducible unitary representation {tt,H) of G let Hk denote the space 
of if -finite vectors. Hk is a finitely generated Z^(g)-module and we define the 
Gelfand-Kirillov dimension of tt by 

GK-dim(7r) := GK-dim(?^A'). 

5.2. Associated variety and nilpotent orbits. The Gelfand-Kirillov dimension 
of an irreducible unitary representation tt is closely related to the associated variety 
of TT. Let us recall the definition of the associated variety. For a finitely generated 
W(0)-module W we again choose a finite-dimensional generating subspace Wq C W 
and define a filtration W ~ ^kLo^k of W by Wk ■= hlkiQ)Wo. The graded algebra 
grW(g) = ®fcLo^'t(0)/^fc-i(0) isomorphic to ^(gc) and therefore we can regard 
the corresponding graded module giW ~ ®fcLo ^k/Wk-i as a module over S{gc)- 
Denote by J := Ann5(gp)(gr W^) the annihilator ideal of grW^ in 5'(gc). The affine 
variety V(J) C corresponding to the ideal J C S{qc) = C[g£] is called the 
associated variety of W. It does not depend on the choice of the generating subspace 
Wo. 

We define the associated variety V{tt) of an irreducible unitary representation 
{tt,H) to be the associated variety of the Z^(g)-module Hk. The variety V{tt) is 
always a i^c-stable closed subvariety of p^. consisting of nilpotent elements and 
hence the union of finitely many nilpotent /iTc-orbits. The relation to the Gelfand- 
Kirillov dimension of tt is given by 

GK-dim(7r) = dimV(7r). 

Assume that G is non-Hermitian. Then there exists a unique non-zero nilpotent 
A'c-orbit O^-^f^ C of minimal dimension m(g) (see |261 Proposition 2.2]). If gc is 
a simple complex Lie algebra then there is also a unique minimal nilpotent coadjoint 
orbit O^'f,-! C gj. By a result of Okuda [33] the two orbits are related as follows: 

Proposition 5.1 (see [33]). Let q be a non-Hermitian simple real Lie algebra. 
Then exactly one of the following three cases occurs: 

(1) gc is a simple complex Lie algebra and Oj„f„ H p^ = C^f^. In this case the di- 
mension of C^f„ is half the dimension of O'^^ and hence given by the following 
table: 



gc 




Bk {k > 2) 


Cfc 


Dk 


0^ 


fC 








idimO,Vi-„ 


k 


2fc- 2 


k 


2k -3 


3 


8 


11 


17 


29 



(2) gc o, simple complex Lie algebra and O^^f^ ^ Pc ~ ^- T^i^ happens exactly 
for g in the following list 






su*(2fc) 


so(fc + 1, 1) 




e6(-26) 


f4{-20) 


dimO«f„ 


4fc- 4 


k 


2{p + q)-l 


16 


11 



(3) g = f)c is the complexification of a simple real Lie algebra f). In this case 

diniO^, = 2,7i([)). 
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In fact, as shown by Okuda [33], in case (2) the orbit O^^^ is the intersection 
of the nilpotent coadjoint orbit Gc ■ Cmin — Be with p^. This orbit is the unique 
coadjoint orbit of minimal dimension in which has nontrivial intersection with 

(or equivalently with g*). 

A direct consequence of this is that for every infinite-dimensional irreducible 
unitary representation (tt, H) of G we have 

(5.1) GK-dim(7r) > m(g). 



5.3. Small representations. Using these observations we can now determine the 
associated variety of some unitary representations constructed in Section 21 For 
this we first calculate their Gelfand-Kirillov dimension. 

Proposition 5.2. (1) Let X be an arbitrary symmetric R-space. For ly Cz iR we 

have 

GK-dini(7r^) = n = dim(n). 

(2) Forg = e6(6), = e(C), g = so(2r+ 1, 2r + 1) and g = so(4r + 2, C) letW be the 
unique unitary subrepresentation of I{v) with K -types W = ®J^^o - . 

Then 

GK-dim(M^) m(g). 

Proof. (1) This follows directly from the more general statement |431 Lemma 6.5]. 

(2) The strategy for the other cases can be described as follows: First calculate 
the dimension of the A'-types - as a function of m and determine the 

highest power D of in, i.e. dim - = cm^ + o{m^) with c 7^ 0. This 
can be done by using the Weyl dimension formula 



(5.2) dimF("'° °) = W 



(A(rn,0,...,0) + P 



a) 



Q6A+(tc.llc) ^ ' 

Here t)c C is a split Cartan subalgebra such that f)c = (f)c H trie) © tc with t 
the maximal torus in nt from (jl.Sp . where A+(Jc, f)c) a set of positive roots for 
A(fc, f)c)j compatible with the ordering of t^, and p'' denotes half the sum of 
all positive roots. Next observe that if Wo = V'^ generates a subrepresentation 
W C I{v) with isT-types l/(™^o,...,o)^ m e No, we have 

k 

dimZ^fc(g)Wo < dim0 Ft^^o--") = c'k^+^ + o{k^+^) 



m=0 

with c' 7^ and hence 

GK-dim(l^) = hm 'o^^^^^i^^^o < n,, ^ n + 1. 

k^oo log k fc-i-oo log k 

This gives an upper bound for the Gelfand-Kirillov dimension of W . Compare 
this upper bound with the lower bound (|5.ip to find that GK-dim(Ty) = D+1 = 
m(g). 

(a) g = ee(6). The Lie algebra i = sp(4) is of type C4. Its corresponding 
symmetric subalgebra is given by m = 5p(2) -|-sp(2). Write t = m© n{ with 
tif as in (|1.3p and choose a split Cartan subalgebra [)c ^ ^C- We can label 
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the roots A(£c, f)c) = {^^^ | 1 < j < fc < 4} U {ie^ 1 1 < j < 4} such 
that = C(ei + £2) + C(e3 + £4) and 

£1 +£2 +£3 + £4 = 271, 
£1 + £2 - £3 - £4 = 272, 

where 71 and 72 were defined in Section fl.il As positive roots we may take 
A+(«c, f)c) = I 1 < j < fc < 4} U {£j I 1 < j < 4}. Then the highest 

weight A(„i 0) — "*7i of V'^™--^^ is given by 

£1 + £2 + £3 + £4 
A(m,o) — m 2 • 

In the Weyl dimension formula (|5.2p the only terms contributing a power 
of m to the dimension occur for a equal to 

£, a = 1,2,3,4), £l±£^i (1 < J < fc < 4). 

Counting these roots yields D = 10. Comparing with m{g) gives by Propo- 
sitionO (1) that D + l = n = m{Q). 

g = e6(C). Note that (£, m) — (e6,so(10) + R) is a Hermitian symmetric 
pair. Therefore, the roots 71 and 72 defined in Section 11.11 form a maxi- 
mal system of strongly orthogonal non-compact roots. In the notation of 
Freudenthal-de Vries [8] we have 

2 \ / 



u 2 3 2 ly ' yi 1 1 1 1 

Let A+({c,t)c) denote the positive roots of te as in [H Table B]. Then 
^{m.o) = "Z7i is the highest weight of Since all roots in A+(tc, ()c) 

have the same length and the angle between two simple roots is either ^ 
or ^ a positive root a contributes a power of m to the dimension in 
if and only if 

^ a 

with a 7^ 0. (In fact, only a = 1,2 occur.) Counting these roots using [HI 
Table B] yields D — 21. Comparison with m(g) gives by Proposition l5.1l f3) 
that D + 1 = 22 = m(0). 

= so(2r+l,2r-|-l). The Lie algebra* = so(2r + 1) +so(2r + 1) is of type 
BrXBr- Its corresponding symmetric subalgebra is given by m = so(2r-|-l). 
Write 6 = m © rij with nt as in (|1.3p and choose a split Cartan subalgebra 
()C C Jc- We can label the roots {±£i 1 1 < i < 2r} U {±£i ±ej\l < i < j < 
roTr + l<i<j< 2r} such that = X]i=i '^{^i + £i+r) and 



7i: 1 < i < 



where 7^ were defined in Section 11.11 As positive roots we may choose 
A+(«c, f)c) = {ei 1 1 < « < 2r} U {£, ± £j 1 1 < i < j < r or r -h 1 < i < j < 
2r}. The highest weight \(m,o....,o) = ^li of giyg^ 

\mfi,...fi) = m{ei + £r+l)- 
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In the Weyl dimension formula (|5.2p the only terms contributing a power 
of m to the dimension occur for a equal to 

ei, ei±ei {2<i<r), Sr+i, Sr+i ± Si {r + 2 < i < 2r). 

Counting these roots yields D ~ ir — 2 and by Proposition 15.11 (1) we find 
L' + l==4r-l = m{Q). 
(d) g = so(4r + 2,C). The Lie algebra i = so(4r + 2) is of type D2r+i- Its 
corresponding symmetric subalgebra is given by m = u(2r + 1). Write 
6 = m © n{ with nc as in (|1.3p and choose a split Cartan subalgebra ()c C 
tc- We can label the roots {±6; ± 1 1 < i < j < 2r + 1} such that 
tc = El=iC(e2..-i +£2,) and 

e2i-i + £2j = i<i<r, 
where 7^ were defined in Section 11.11 As positive roots we may choose 
A+(tc, f}c) = {e* ± £j 1 1 < i < j < 2r + 1}. Then the highest weight 
A(m,o,...,o) = "171 of y("''0---.o) is given by 

A(m,o) = +£2)- 

By the Weyl dimension formula (j5.2p the only terms contributing a power 
of m to the dimension occur for a equal to 

£1+62, £1 ±e, (3 < i < 2r+ 1), £2 ± (3 < i < 2r + 1). 

Counting these roots yields D — 8r ~ 3. Comparing with ?Ti(g) gives by 
Proposition [5T] (31 that L» + l = 8r-2 = 771(0). □ 

Theorem 5.3. Let W be one of the following representations: 

(1) = s[(l + .s,R),s[(l + s,C),s[(l + s,H),e6(_26)- LetW ^ I{v) with v e iM., the 
unitary principal series associated to the corresponding non-unital symmetric 
R-space of rank 1, 

(2) Q ~ 06(6), e6(C),so(2r + 1, 2r + l),so(4r + 2, C); Let W be the unique unitary 
subrepresentation of L{iy) with K -types W — ®^^q ' ■ 

Then the associated variety of W is the closure of the minimal nilpotent Kc-orhit: 

In particular, the Gelfand-Kirillov dimension of W attains its minimum among all 
infinite- dimensional irreducible unitary representations of G and any of its covering 
groups. 

Proof. First note that O^f^ is the unique coadjoint orbit with smallest possible di- 
mension that can occur in the associated variety of a non-trivial irreducible unitary 
representation. We show that the associated variety of the representation in ques- 
tion has in all cases the same dimension as O^f^ and hence the result follows. Since 
the dimension of the associated variety is equal to the Gelfand-Kirillov dimension 
of the representation it suffices to show that the Gelfand-Kirillov dimension coin- 
cides with the dimension m{g) of O^fi,- For case (2) this is already explicitly stated 
in Proposition 15.21 For case (1) the statement in Proposition 15.21 (1) reduces this 
to show that dimn = rn{g) which we check for the separate cases: 

(a) Q = s[(l + s,M), n = M(l x s,M). By Proposition O (1) the orbit has 
dimension s and hence dim 0^f„ = dim n. 
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(b) 9 = sl(l + s, C), n = A/(l x s, C). We have dim0^f„ = 2s by PropositionlO (3) 
and hence dim 0^f„ = dim n. 

(c) 9 = + 5,11), n = Af(l X s,H). Since g = su*(2s + 2) we have by Proposi- 
tion O (2) that dimO^Jj^ = As = dimn. 

(d) g — ^6{~26), n = M{1 X 2,0). Again by Proposition 15.11 (2) we obtain 
dim off = 16 = dimn. □ 



mm 



Remark 5.4. For the groups G ~ S0(2r + l,2r + 1) and G = £^6(6) it is hkely 
that the irreducible unitary representation W in Theorem 15.31 is the minimal rep- 
resentation, i.e. the annihilator of W in ^(g) is equal to the Joseph ideal. In both 
cases the complex Lie algebra gc is simple and not of type A. Hence the Joseph 
ideal is the unique completely prime two-sided ideal in W(g) with associated variety 
equal to the minimal nilpotent coadjoint orbit in gj by |22l Proposition 10.2] (see 
also [101 Theorem 3.1]). From Theorem 15.31 it follows that the annihilator of W in 
Z^(g) has associated variety equal to the minimal nilpotent coadjoint orbit in g^. 
Therefore it remains to show that the annihilator is a completely prime ideal in 
U{9). 

Appendix A. Proof of Proposition 14.31 

This section is devoted to the proof of Proposition 14.31 For this we put 
| + 1,/3:=| 



2(TOfc -f pfc), a ;= I + 1, /3 := I -I- e and 7 := d. Then by Lemma [^?T] 



C(m) = 1 - ^d[x\ 
2n 

where we abbreviate 

r 

d{x) :— (a(a;, k) 4- b{x, k)), 

k=l 

^ ~ Xk (Xk + 1) • l{ xl-x- ' 

Xk [xk-l) fj-^ xl ~ 2;2 
Proposition 14.31 now follows from the following identities: 

Proposition A.l. Let a; G C with Xj ^ Xk for j ^ k and Xj 7^ ±1. Further let 
a, /?, 7 € C. 

(1) For 7 0, 2 we have 

(2) For 7 e {0, 2} we have 

d(x) = 2r-2(l-a)(l-/?)^— ^. 

fe=i ^ 
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Proof. For convenience, we set r := {1, . . . , r}. For fixed fc G r we first note that 

^ {x,,±jf^x^^ = TT (^1 I ^(^^^^^-) ^ = V ^''"(^^^^fc)'^' 

Then a short calculation shows that 

^ a(x, fc) + fc) = ^ ^ — ^ — ^ ^l-^l^ ''^ 

k=l k=l JCr\{k} - ^JlljGji^k - ^j) 

where 

Pm{x) := - {{x + a){x + /3){x - 1)(7 + 2a;)" + (x - a){x - /3)(x + 1)(7 - 2a;)™). 

X 

We note that p„i is an even polynomial of degree < m + 2. Changing the order of 
summation by using the bijection J h->- Ju {k} between subsets of r not containing 
k and those containing fc, we obtain 

\ P\J\-i{xk) 



JCr. keJ 



i4-^)U,eJ\{k}i4~^'y 



In order to evaluate the inner sum, we set a;o := 1 and Jq := JU {0}, and find that 

P\J\-lixk) - ( \^ P\J\-liXk) \ P\J\-l{xo) 



fee,/ 



i^l-'^)UjeJ\{k}i^k-^'j) \k^j^UjeJo\{k}(^k- ^'j) J Ujej(^o- ^'j) 

Due to the following lemma (applied to {yi, . . . ,y\jg\} = {xj \j e ^^o}); the first 
term vanishes if \J\ > 1. Therefore, 

Resubstituting a;o by 1, we obtain for the first term by an elementary calculation 

k=l ^ k -^0 -^0 -^fe/ ^.^i 

Using |J,n(l) = 2(1 — a)(l — f3){j — 2)™ in the second term, we find 

JCr ilje-''^^ ^j) llfe=iU a;^J jer\j 

Now if 7 e {0, 2} only the summands for \J\ = 1 survive and the claimed formula 
follows. For 7 7^ 0, 2 we find 

llfe=iU ^k) JCr jer\./ 

^ 2(l-c.)(l-/j) / Ejc.(7(7-2))l-^ln,-g.v(l-^|) \ 

7(7-2) I + nLi(i-4) J 
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Finally, since 



;(7(7 - 2))i^i n (1 - = n (^(^ - 2) + (i - 4)) = n((^ - 1)' - ^')' 

JCr jer\J k=l k=l 

it follows that 



7(7 - 2) \ ^^^^ 1 - X 



This completes the proof. □ 
Lemma A. 2. Let yi, . . . , be pairwise distinct real numbers. If m < N —1, then 



y = 0. 



Proof. Let 



4(x) n ^ 



be the fc'th Lagrange polynomial. Then, (kivj) = Skj and hence 

span{4|fc = l,...,iV}=R[X]<^_i, 
the space of polynomials of degree < iV — 1. For any p G R[X]<jv_i we obtain 

AT 

p{x) y^p{yk)h{x). 

k=l 

Applied to p{x) — a;"'+^ with to < — 1 and evaluated at x = 0, this yields 

N „, N 

We may assume that yj ^ for all j and conclude the statement. □ 
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